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NETWORK USAGE MONITORING DEVICE 
AND ASSOCIATED METHOD 



FIELD OF THE INVENTION 
The present invention relates generally to network usage monitoring and, 
more particularly, to a network usage monitoring device and associated usage 
monitoring methods that operate to capture usage data at a point of network access. 

5 

BACKGROUND OF THE INVENTION 
Communication networks, such as the Internet, corporate intranets and the 
like, have become the chosen form of iafoimation distribution in the workplace, as 
well as, fbe home. With the need to access information fix>m such networks 

10 increasing, so to is the need to provide a means for monitoring the information 
distributed by such communication networks. Monitoring of this nature 
heretofore referred to as "usage monitoring", can provide valuable information, 
statistical or otherwise, to network service providers, network users or network 
beneficiaries, such as network advertisers. 

1 5 The network service providers can benefit from usage monitoring in many 

difierent applications; for instance, it will provide the capabihty to create network 
user profiles based on the information that the user has accessed and the 
information that the user has provided to the network. User profiles are beneficial 
to the service provider in many applications, such as, intelligently caching data (i.e. 

20 storing copies of network pages accessed by the user) and/or prefetching data (i.e. 
predicting fiiture network page access and storing such). Additionally, usage 
monitoring information is instrumental in case of a network security breach. It 
provides valuable post-breach information that can be used to trace the history of 
Ihe breaching party. 

25 Communication networks, such as the Intemet, have provided another 

medium over which advertisers can distribute information and solicit customers. 
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Because of the vast numbers of people who rely upon the Internet on a frequent, if 
not daily, basis, advertisers have invested enormous amounts of money to deliver 
advertisements via the Internet Unfortunately for advertisers, advertisements that 
are merely distributed on a mass basis are not typically cost effective since many, 
5 if not the majority, of the recipients have no interest in the particular product or 
service being advertised. As such, advertisers commonly work to target the 
advertisements to a subset of people who are believed to be most interested in the 
product or service being advertised. For example, new homeowners may be most 
interested in security systems and, as a result, residential security companies may 
10 target their advertising to the new homeowners. To be able to target their potential 
customer base, the advertisers must be able to access usage monitoring data in a 
real time fiishion and be provided with data that is formatted, or can be readily 
formatted, to address the demographics to which their particular product is 
addressed. 

15 Additionally, usage monitoring information can be important to Content 

Distribution Networks (CDNs) as they serve to optimize their distribution network 
on the basis of usage information and provide better services to their clients and 
the subscribers. 

Network users, such as households or corporations, can benefit firom usage 
20 monitoring information by monitoring the usage of mdividuals within the 

household or the corporation. Such monitoring insures that the individuals are not 
accessing inappropriate information or spending an mordinate amount of time 
browsing the Internet or intranet. 

While conununication networks, such as the hitemet, provide a medium 
25 over which an enormous number of people communicate, network service 

providers, network beneficiaries and the users themselves have generally been 
unable to determine tiie exact nature of the users, the content of the information 
accessed by the users, the demographic breakdown of the users and o&er usage 
related information. For example, network advertisers have had difficulty 
30 detOTnining, in real-time, both detailed demographics regarding the users and 
information relating to the actual content requested by the us^. Network service 
providers have had difficulty tracking, in real time, the information accessed by 
any one user so as to provide more intelligent caching and prefetching capabilities, 
security breach investigation capabiUties and &e like. Additionally, the network 
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users (i.e. households or corporations) have not been provided with an adequate 
means of tracking the use of the individuals (i.e. family members or employees) in 
terms of content accessed or duration of the access session. 

In this regard, current techniques exist for monitoring network usage at 
5 different discrete locations within a conmiunication network. For example, 
network servers and/or routers located within the stream of network 
communication can monitor usage by keeping logs of access requests for the 
content hosted by the server. These logs provide mformation about the number of 
users that access a particular page or a set of pages and the navigational sequence 

10 through the pages. Software is also provided for performing statistical processing 
of the server logs. Likewise, usage can be monitored at the user level, i.e., at an 
individual's computer, by keeping, in memory, a log of the content accessed by the 
user. This log provides information about the usage patterns of the individual user 
including the content accessed, the navigational sequence through the content and 

15 the amount of time spent on each page. Further, some Intemet service providers 
(ISPs) can include a proxy-caching server that stores the most frequently requested 
information. These proxy-caching servers can be designed to monitor tiie web 
content usage of those users that are routed to the network via the proxy-caching 
server. Uiifortuiiately,pTOxy-cadiing servers requke user inte^ 

20 network administrator or the like in order to appropriately configure the proxy 
cache. Further, not nearly all client requests are directed to the proxy-caching 
server, thereby rendering the usage data incon^lete and much less valuable. 

As will be noted, each of these prior techniques is associated with and 
monitors the usage of a discrete device or portion of the web. For example, 

25 monitoring that is conducted at the server level and at the user level obviously only 
identifies usage occurring at the server and by the user, respectively. In addition, 
monitoring conducted via a proxy-caching server only identifies some of the cUent 
requests for a particular ISP network. While these various types of usage 
monitoring are of some assistance to network service providers, network 

30 beneficiaries and network users, these entities desire usage monitoring on a much 
broader scale covering many different users attempting to access information 
hosted by many diffident servers, potentially on the networks of many different 
service providers. 
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SUMMARY OF THE INVENTION 
A network usage monitoring module is therefore provided for monitoring 
network usage at a network access point, i.e. network traffic aggregation point, 
typically at a gateway device or a similar network interface device. As such, the 
5 network usage monitoring module ofthe present invmtion can monitor the usage 
of a number of network users who are attempting to access various netwoik 
services provided via the gateway device. Thus, the usage information collected 
by the usage monitoring module of the present invention is ccmsiderably more 
robust than that offered by conventional monitoring techniques. As such, the 
1 0 infbhnation is considerably more valuable to network service provide, netwoik 
users, netwoik beneficiaries and the like. In addition, the usage monitoring method 
and apparatus of the present invention offers a number of particular features to 
improve the monitoring process as well as the value of the usage information that 
is collected. 

IS Jn. one embodiment, the usage monitoring method and ^paratus captures 

all bytes of data transmitted between user and network services and records usage 
information associated with netwoik addresses (i.e.. Uniform Reference Locations 
(URLs)). TypicaUy, the usage monitoring method and {^paratus will limit the 
storage of information to instances where a user gains access to the netwoik 

20 service and discards any usage information associated with addresses that are 
determined to be inaccessible, thereby reducing the size of the resulting database 
and improving subsequent searching of the database and caching of the database. 
The usage monitoring method and apparatus also pennits the monitoring of 
streaming content in real time and facilitates the implementation of web-based user 

25 surveys. The usage monitoring method and apparatus of the present invention 

further allows for the monitoring of user query strings as well as information input 
by a user into a web form. 

Importantly, the usage monitoring method and apparatus of the present 
invention c^tures navigational sequences that include one or more designated 

30 network addresses. In this regard, the designated address can either be designated 
in advance or determined on the fly by the usage monitoring module to be the most 
popular addresses. Alternatively, the usage monitoring method and tpparatus can 
monitor the addresses that are in a neigihborhood of a designated netwoik address 
regardless of the sequence or order in which the addresses are accessed. In a 
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similar fashion to the technique by which flie usage monitoring method and 
apparatus of the present invention monitors ttie addresses in the neighboifaood of 
the designated address, the usage monitoring method and apparatus can monitor 
the domains that proceed and/or follow a designated domain. Furthermore, the 
S usage monitoring method and apparatus can monitor an associated address that is 
in ttie neighborhood of a designated address, i.e., an associated URL that has 
proceeded or followed a designated address by no more than a predetennined 
number of addresses, regardless of the particular navigation sequence. The 
associated address can be defined in many ways and is typically defined by a user 

10 or network administrator to identify other associated addresses. 

By csqpturing a more robust set of information related to the usage of the 
communication network, the usage monitormg method and apparatus of the present 
invention can provide very useful mformation to usage monitoring information 
clients. For example, the usage monitoring method and apparatus can include 

1 5 detailed user demogr^hics and information relating to die content of the accessed 
sites or of native streaming media. In addition, by monitoring the usage at the 
gateway to the Internet, i.e., the point of traffic aggregation, the usage monitoring 
method and apparatus can monitor the usage of a large number of users requesting 
access to many different service providers. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of the usage monitoring apparatus according to 
one embodiment of the present invention. 

Figure 2 is a block diagram of the usage monitoring apparatus according to 
another embodiment of the invention. 
25 Figure 3 is a flow chart illustrating the general operations performed by the 

usage monitoring method and sqpparatus of one embodiment of the present 
invKition. 

Figure 4 is a flow chart illustrating the navigational sequencing operations 
performed by usage monitoring method and apparatus, in accordance with another 
30 embodiment of the invention. 

Figure 5 is a diagram illustrating a relational database that could be utilized 
by one embodiment of the usage monitoring method and apparatus of the present 
invention. 
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DETAILED DESCRIPTION OF THE INVENTION 
The present invention now will be described more fully hereinafter with 
reference to the accompanying drawings, in which preferred embodiments of the 
invention are shown. This invention may, however, be embodied in many different 
5 forms and should not be construed as limited to the embodiments set forth herein; 
rather, these embodiments are provided so that tiiis disclosure will be borough and 
complete, and will fully convey the scope of the invention to those skilled in the 
art like numbers refer to like elements througjiout 

A network 10 impl^enting usage monitoring, in accordance with the 

10 present invention is depicted in Fig. 1 . The usage monitoring module 12 is 

preferably implemented in or disposed in communication with a gateway device 
14. For example, in an embodiment in which the usage monitoring module is 
implemented in a gateway device, the gateway device can be a universal subscriber 
gateway provided by Nomadix, Inc. of Westiake Village, California and described 

15 in U.S. Patent AppKcation Serial Nos. 08/816,174, 09/458,602, 09/458,569, 

09/541,877 the contents of which are incorporated by reference as if setforth fully 
herein. Alternatively, tiie gateway device can be any of anumber of other gateway 
devices csqpable of inq>lementing programming modules as known to those of 
ordinary skill in the art. It should be noted that while implementation in 

20 conjunction with a gateway device is preferred, the usage monitoring module could 
be implemented with other devices or be an indq>endent device so long as the 
module or device implementing tiie module is positioned in the network so that it 
receives all requests/queries &om a plurality of users and receives all responses 
coming from all the network services that comprise the network (i.e., a point of 

25 network traffic aggregation). In one embodiment of the invention, the gateway 
device implementing the usage monitoring module will be positioned within the 
network between a plurality of user devices 16 and the remainder of the 
communications network (i.e. router 18; service provider complex 20 and Internet 
22). Thus, the usage monitoring module is in[iplemented at a network point of 

30 access that serves as a point of traffic aggregation for all network services that are 
included within the conununications network. 

The network users can communicate with the gateway device via a number 
of dififerent user devices 16, such as handheld communication devices, portable 
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commimication devices (i.e. laptops), personal computers or Ihe like, which are in 
communication with the gateway device via an access multiplexor 24. 
Additionally, depending on the means of conmiunication between the user and the 
netwoik, the network architecture may mclude a network interface device (NID) 
5 (not shown in Figure 1) that serves to link the various means of communication, 
such as DSL, cable, LMDS (Local Multipoint Distribution Services), dial-up, 
leased line, wireless or tiie like. While Figure 1 depicts the communication 
net^oik users communicating with the network tbrougih a single access 
multiplexer, in a typical communication network multiple access multiplexers will 

10 be implemented, as well as multiple netwoik iaterfBce devices, to allow for 
expanded user capabilities. 

The gateway device 14 typically pemiits the users to conmiunicate with a 
variety of IP network services, offered either by the ISP (Internet Service Provider) 
or an mterprise network (i.e. corporate intranet or the like). As shown in Figure 1, 

15 the gateway device may be in communication with router 18 that serves to route 
network communication to the service provider server complex 20, the Intemet 22 
or any other network service (not shown in Figure 1). While Figure 1 depicts a 
single router, in a typical communications network, multiple routers and/or 
switching devices will be in communication with the gateway device for properly 

20 routing network communications to their addressed destination. 

According to the present invention, the usage monitoring module captures 
all packets transmitted between the users and the network services. Filtering is 
performed to extract all relevant packets that the usage monitoring client prefers to 
monitor (such as HTTP requeste and responses, DNS requests and responses, etc.) 

25 and forwards such to an associated usage monitoring database 30. The requests by 
a user for network services and the responses from the network sorices are 
C25)tured by the usage monitoring database. The usage monitoring database can be 
either internal to the device implementing the usage monitoring module or external 
to the device so long as the usage monitoring database is communicably connected 

30 to the usage monitoring module. In one embodiment of the invention, the usage 
monitoring database stores information associated witii the HTTP requests and 
responses and processes the information as described hereinafter. 

Figure 2 depicts a schematic representation of a netwoik implementing 
usage monitoring having a series of databases, in accordance with an embodiment 

-7- 
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of the present invention. As a result of its position at a point of trafGc aggregation 
in tiie network architecture, the gateway deAdce 14 receives requests from a nuniber 
of subscribers for Internet or other network services. In this regard, the usage 
monitoring module implemented in the gateway device is capable of capturing all 
5 of the packets since the gateway device operates in a promiscuous mode. In tibie 
Figure 2 embodiment the usage monitoring database is implemented by a usage 
monitoring database server 32 that has access to auxiliary databases associated 
witii the usage monitoring process. 

The usage monitoring module 12 will capture packets and route them to 

10 their respective database, in accordance with one aspect of the present invention. 
If the captured packet represents a request or query from the user, the request is 
stored in the request database 34. In addition to capturing user request packets, the 
usage monitoring module will capture packets representing responses and other 
messages transmitted by the network services. In this regard, the usage monitoring 

1 5 module will match the responses with their respective requests. If the user request 
or query is successfril, i.e. the network service grants access or otherwise responds 
to the query, then the response is matched with tiie user request and the 
information pertaining to this matched set of request/response is forward to either a 
temporary database 36 or the primary database 38. If the response from the 

20 network service indicates that a valid response is not forthcoming, such as access is 
denied, access is unavailable, a service error has occurred, etc. or if a time-out 
occurs before a response is received, then &e request database deletes the 
associated request from the database. In a similar fashion, if the response Gxm the 
network service can not be matched with a user request or if the response occurs 

25 outside of a predetermined allowable time window, then tiie response from the 
network service is discarded. This aspect of the invention, which provides the 
csqpability to discard requests that can not be matched with valid responses and vice 
versa, allows the usage monitoring apparatus to conserve storage, thereby 
minimizing the storage space required to support the usage monitoring s^aratus. 

30 By not permanently storing requests/queries that do not result in successfril 

responses, the prinuoy database 38 can intelligently provide superior performance 
in the searching/caching operations. 

Once file usage monitoring module has matched the user request to the 
network service response, the roformatibn found in the requests and responses is 

-8- 
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Stored in either the temporary database 36 or the primary database 38. The usage 
monitoring module will determine if the user request is one of a pre-determined 
number of initial requests to a particular network service, such as a web page. If 
the determination is made that it is within an initial request or the pre-determined 
S number of initial requests, then the information found in the request and the 
response is stored in the temporary database. Once the netwoik service has been 
accessed more than the pre-determined number of times, the information is 
transferred to and stored in the primary database 38 and subsequent 
requests/responses are proxied to and stored in the primary database. The 

10 inq)lementation of the temporary database accounts for the fact that many network 
services are only requested a single time or a minimal amount of times and, 
therefore, the statistical presence of these minimal use requests/responses may not 
be required when processing of the usage data information occurs. In this regard, 
the primary database is minimized so as to be more accommodating to subsequent 

15 processing of the database by eventual usage monitor information chents (i.e., 
network service providers, network bmeficiaries or network users). The usage 
monitoring module can be configured so that the predetermined number of initial 
requests is selected to balance the competing goals of recording all activity of a 
user and streamlining the database. In some cases, however, the predetermined 

20 . numberissetto 1. ' 

Additionally, the netwoik usage monitoring q>paratus of the present 
invention may include a database of designated network addresses 40. The 
designated netwoik addresses are typically frequentiy visited netwoik addresses 
tiiat are predefined by the netwoik usage monitoring administrator, typically the 

25 gateway device administrator or the like. The designated network addresses are 
most often addresses that the usage monitoring information cUent deems 
particularly important. The usage monitoring module allows for navigational 
sequences to be stored in the database of designated network addresses for those 
network addresses predefined as "designated". A navigational sequence will 

30 typically include a predefined number of immediately preceding and/or 

immediately succeeding network addresses accessed by the user. For eicample, a 
netwoik beneficiary, such as a commercial advertiser, may desire Hhat a particular 
Mtemet site be designated for usage monitoring purposes. As such, when a 
network user accesses the designated Memet site, the usage monitoring module 
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wiU recognize ttie network address as being designated and will record and store 
the preceding and succeeding navigation sequence leading up to and following the 
access of the designated network address. It should be noted that die navigational 
sequence could extend beyond a single user session provided the int^val between 

5 sessions is within a predetermined session window. 

Jn one embodiment of the invention the usage monitoring apparahis is in 
communication with a processmg engine 42 (i.e. database front-end), such as a PC 
or the like. The processing engine is in conomujrication with the usage monitoring 
database and accesses the database in conjunction with a GUI (Grsphical User 

10 iiterface) implemented on the processing engme. The processing engme accesses 
the database in accordance with the commands of the usage monitoring client to 
process the usage monitoring data according to specific, user requirements. In one 
embodiment of the present invention the processing engme is in communication 
with an insertion server 44 (shown in Figure 2). For example, the insertion server 

15 may comprise a targeted advertisement insertion server or a targeted survey 
insertion server. The processing engine implements the necessary commands to 
determine what advertising or survey would benefit the network provider or the 
network beneficiary based upon collected usage monitoring data. Once a targeted 
advertismient or survey determination is made a command is sent to the insertion 

20 server and an appropriate advertisement or survey is forwarded to the network user 
via communication with the gateway device 14. The advertisement or survey will 
typically be in the form of apop-iq) control panel sent to the user's communication 
device. 

Figure 3 illustrates a flow diagram of a method for usage monitoring, in 
25 accordance with an embodimmt of the present invention. At step 100, the usage 
monitoring module, typically implemented on a gateway device, reads all the bytes 
of information that are being transmitted from the plurality of users who access the 
network via the gateway device and tihie bytes coming from the network services. 
The gateway device will typically operate in a promiscuous mode to insure that all 
30 bytes of information, regardless of destination address, are processed by the usage 
monitoring module. At step 110, the usage monitoring module detects the start of 
an HTTP (HyperText Transfer Protocol) header in the TCP (Transmission Control 
Protocol) flow. 



-10- 
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Hie usage monitoring module may be optionally designed so as only to 
monitor the usage of network users who have accessed more than a predetermined 
number of different network addresses, such as Internet URL's (Uniform Resource 
Locator) during a session. This provides for a reduction in the database without 
5 significantly impairing the integrity of the data because users who only access a 
minimal number of addresses may not provide vital statistical data to the usage 
monitoring client Therefore^ at optional step 120, a detemiination is made as to 
whether the current session flow for tiie user has exceeded the predetermined 
threshold minimum. If the user has not accessed more fbm the predetomined 

10 number of URL's during a session, the usage monitoring module contmues to 
monitor, at step 110, the requests transmitted by the user, but does not further 
analyze and store the requests until the number of requests exceed the 
predetermined threshold 

Once the user has issued more than the predetermined number of requests 

15 during a session or if the usage monitoring.module does not implement this 
optional filter, then, at step 130, the usage monitoring module determines if the 
packets represent a request or a response, typically by analy2dng the HTTP header 
of the packet. If the captured packet represent a request, then at step 140, the 
request is stored in the request database. If the cs^tured packet represents a 

20 response, then at step 150, the response is checked to determine if the status code 
in the response is a valid or invalid status code. 

Usually, while monitoring network service accesses or other requests, the 
&ct that the user might not havelseen able to access the service for some reason (it 
may have been a dead link or the server may have been down temporarily) is not 

25 taken into account. In one aspect of the invention, however, the response status 
codes fix)m the requested network service are also monitored in order to record 
only the addresses of the network services that the user actually gains access to 
with the other requests being deleted from the database. This can result in a 
significant saving in storage and processing computation. 

30 The status code is a three-digit integer number returned by the requested 

server. Tlie first digit ofthe status code represents the class of die response. Some 
of the more common status codes returned are: 

1) 200OK 

2) 301 Moved Permanently 

-11- 
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3) 304 Not Modified 

4) 401 Unauthorized 

5) 403 Forbidden 

6) 404 Not Found 

5 7) 500 Ihtemal Server Error 

8) 501 Not Implemented 

While a status code of 200 indicates fliat the web page can be visited, the 
other status codes indicate that the web page can not be visited. If the response 
status code is determined to be invalid, i.e. a status code other than 200, then, at 

10 step 160, the response is discarded and the matching request is deleted i&om the 
request database. Additionally, a user request may be deleted firom the request 
database if a predetermined timeout period is exceeded prior to receiving an 
associated network response. If the response status code is determined to be valid 
then, at step 170, the network response is matched witii the associated user request 

15 in the request database. If no match is found for the returned network response 
such as in instances in which an excessive amoimt of the lime passed between the 
request and the response, then, at step 180, the network response is discarded. 

At step 190, the usage monitoring method and apparatus optionally 
determines if flie matched requestA:esponse is in the primary database or the 

20 temporaiy database. While all requests/responses and information associated with 
such can be stored in the primary database, the usage monitoring method and 
apparatus of one aspect of the present invention initially, and optionally, stores 
data in a temporary database if the requested network service has been requested 
for the Jfirst time or, if configured as such, requested for less than a predetermined 

25 number of times. Thereafter, the request/response information is transferred to 
and/or recorded in a primary database once the network service is requested for 
more than the predetermined number of times. As previously discussed, since 
many of the network addresses, e.g. Internet URL's, are only requested a single 
time the size of tiie primary database can be minimized while continuing to store 

30 aU of the network service requests/responses that are visited more than once by any 
user. This can be accomplished by establishing a separate, albeit tenoporary, 
database for network services that are infirequently requested less than a predefined 
minimum number of times. 
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Thus, if no entry is found in both the permanent database and the temporary 
database (or if the minimum threshold Umit has yet to be exceeded for entries in 
the temporary database) then, at step 200, the request/response info is stored in the 
temporary database. If an entry is found in the permanent database then, at step 
S 210, tiie permanent database is updated to indicate a new request/response entry. 
Additionally, if an entry is foxmd in the temporary database and the current access 
qualifies it for inclusion in the pemianent database (i.e., the predet^mined 
minimum threshold limit is being exceeded) then, at step 210, the entry or entries 
from the temporary database are transferred to the permanent database. 

10 In one embodiment of the invention, the usage monitoring database stores 

the URL of the request and associated response. In general, a URL will include the 
protocol prefix, the port number, the domain name, subdirectory names and a file 
name. In alternate embodiments of the present invention, the usage monitoring 
database will preferably store, in the primary database, a variety of other data 

15 related to the request and the response. In this regard, the following data is 
typically coUected fi'oin the request/response and stored in the ^ropriate 
database: 

a) Subscriber Identification (This identification may be anonymized 
before being provided to the usage monitor client by mappiag it to a 

20 random number and will be indexed to user profiles that provide 

information about tiie user's age, gender, income, etc.) 

b) URL (URL's will be logically arranged in clusters) 

c) Timestamp- indicating the time at which the packet was processed by 
the usage monitoring module. 

25 d) Referring address (obtained from the HTTP header) - typically, the 

address that provided the hyperlink to the accessed address. 

e) Content-type (typically obtained from tiie HTTP header) 

f) Content length 

g) Response Status Code 

30 h) User Query String (usually sent as an argument to a CGI (Common 

Gateway Interface) ^plication or as a result of the networic user posting 
a form). 
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The preceding listing is not to be constraed as exhaustive, other 
information may also be extracted for the requests/requests as is dictated by the 
application for which the inforaiation will ultimately be used. 

While the information included in the user requests and netwoik responses 
5 can be stored in different manners, the primary database of one embodiment of the 
invention is a relational database as depicted in Fig. 5, which relates the users and 
respective addresses/URLs. A relational database is a well-established way of 
storing data sets that may have relationships with each other. In this instance, each 
network user and each accessed address are stored one time only instead of both 

10 items being stored every time a network user accesses the address. Each access of 
a page can then be represented as a node (referred to as a tuple in the context of 
relational databases) that joins the entry of the user accessing the address and the 
entry of the address being accessed. This node will then include additional 
information related to that access/accesses such as time of access, frequency of 

15 access, content length and type, etc. For example, in the instance in which user 1 1 
accesses address/URL 25 the relational database stores user 1 1 in Table 1 and 
address/URL in Table 2 with a node linking the entry from Table 1 and the entry 
from Table 2. This form of data storage provides for ease in data retneval, such 
that, the usage monitoring client can access all the nodes that a user is linked to in 

20 order to CQnq>ile information. ' 

In accordance with the flow diagram of Figure 4, an alternate embodiment 
of the present invention is diagramed that entails usage mraitoring in the form of 
navigational sequence capturing. Network service providers and network 
beneficiaries, such as network advertisers, are interested in knowing not only what 

25 network services are most frequently accessed and how much time is spent 

accessing them but also the navigational sequences that include these frequentiy 
accessed addresses. The network administrator and/or the gateway administrator, 
at the bequest of the sCTvice provide or network beneficiary, may predefine or 
'^designate" specific network addresses for navigational sequencing monitoring. 

30 The administrator will store tiie designated network addresses in the designated 
network address database. The navigational sequence related to a designated 
network address will comprise ai predefined number of network addresses that were 
accessed by the user inmiediately preceding and/or succeeding the access of the 
designated network address. It should be noted that while a typical navigational 
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sequence will occur duiing a single user session, a navigational sequence may 
extend beyond the single user session provided the interval between two sessions is 
under a specified threshold. Additionally, a designated network address may be 
part of more than one navigational sequence; therefore, the usage monitoring 
S module of the present invention may capture and store the following information: 

■ All the navigational sequences the designated network address is a part 
of. 

■ Only the most popular navigational sequences (which exceed a certain 
threshold of visits) that a network address is part of. 

10 ■ The fi-equency of a particular navigational sequence that the designated 

network address is a part of. 
Configuration of the navigational sequence database, typically included in the 
primary database will be at the option of tibie network administrator or gateway 
administrator. 

15 As known by those of ordinary skill in the art, Zipf s Law states that the 

number of occurrences of a word in a sufficiently long stretch of text is the 
reciprocal of the order of the firequency occurrence. For example, the 10*** most 
firequent word occurs exactly 10 times less frequently than the most firequent word. 
While a pure Zipfs law relationship does not readily occur, power laws (where the 

20 relationship between the order of the fi:equency of occurrence and the frequency of 
occurrence is not linear) exist in a wide variety of situations, such as network 
service access firequency. One consequmce of tins is that a small number of 
network addresses accounts for most of tiie user accesses (e.g. 5% of the available 
network addresses account for 95% of network accesses). These assumptions 

25 imply that statistical data, i.e. navigational sequencing and the like, for only a small 
percentage of network addresses (e.g. 5%) needs to be readily captured and stored 
within the usage monitoring database, hi tiiis regard, the network service 
providers and/or the network beneficiaries are provided the option of either 
predefining the designated network addresses that will be navigational sequence 

30 monitored or allowing the usage monitoring module to navigational sequence 
monitor the most firequently accessed network addresses as defined by the 
database. 

Referring again to the flow diagram of Figure 4, the usage monitoring 
module of the present invention monitors the navigational sequences of the 
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designated network addresses in, generally, real-time. It should be noted that steps 
100-170 of the Figure 4 flow are identical to the steps in the Figure 3 flow diagram 
and, therefore, further discussion of these steps will not be necessary. Navigational 
sequence monitoring is accomplished according to the present invention by 
S keeping track of the previous k network addresses accused by each user in their 
current session. The number ^specifies the number of network addresses 
preceding and succeeding the designated network address. The lvalue can be 
specij&ed by the network or gateway administrator and can differ over different 
designated network addresses. Although this description indicates that the same 

10 number of network addresses are monitored both before and after the designated 
network address, a different value can be assigned for the preceding and 
succeeding network addresses, if desired. ' 

At step 220, the usage monitoring module determines whether the network 
user has accessed a designated network address in the previous k network service 

15 accesses. If a detennination is made that tiie network user has accessed a 

designated network address in the previous k accesses then, at step 230, the current 
accessed network address is then added to the navigational sequence being 
constructed. If a determination is made that the network user has not accessed a 
designated network address in the previous k accesses then, at step 240, the usage 

20 moriitormg module determines ifthe current accessed network address is a 
designated network address by attempting to locate an entry in the designated 
network address database. Ifthe usage monitoring module determines that the 
current accessed network address is a designated network address then, at step 250, 
a command is sent to store the previous k network addresses accessed by the user 

25 as a navigational sequence. Ifthe usage monitoring module determines that the 
current accessed network address is not a designated network address then, at step 
260, the current accessed network address is added to the Ust of network 
addresses for that particular user and the oldest network address in the kmax list is 
discarded ifthe i^ax list already has ^ network addresses. The ^ list is stored 

30 because it is not known beforehand which network addresses will be accessed by 
the network user and, thus, the kmax list provides for a listing of preceding network 
addresses ifthe network user accesses a designated network address and a 
navigational sequence is required. 
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In addition to monitoring user access requests and network service 
responses, the usage monitoring method and qjparatus of the present invention can 
monitor usage of streaming content, in accordance with yet another embodiment of 
the present invention. The streaming content can either be live or recorded and the 
5 c£^ture will occur, as the streanodng content is deUvered fiom the native streaming 
media server to the network user via the gateway device. Streaming content is 
typically high volume content so it is not possible or practical to store the 
information related to all streaming content data packets. Therefore, while all 
streaming content packets are captured at the usage monitoring module a 

1 0 significant portion of the packets will be filtered out to accoimt for this high traffic 
volimie. Constant monitoring of ttie streaming content packets will allow for 
significant filtering to occur and will provide information pertaining to the amount 
of data transferred during the streaming coxmection, the time duration of the 
streaming connection, etc. For this constant monitoring to occur the device 

1 5 implementing the usage monitoring module, typically a gateway device, will need 
to allocate a certain amount of state in memory to keep track of the current or last- 
known status of the streaming content. Additionally, the usage monitoring 
module or, alternatively a post processing application, will track users currently 
accessing streaming content, the site &om which the streaming content is coming 

20 fiom, file amount of data transfaxed during a connection and the last time a 

streaming content packet was observed for a particular usei/streaming content pair. 
Since there is no explicit coimection termination in the case of streaming content, 
the usage monitoring module will need to implement a timeout to determine when 
the content steaming terminated. These tracking fimctions can occur at the usage 

25 monitoring module or the unformatted data can be sent on a per streaming packet 
basis to a post processing application to determine the tracking parameters. 

Jn yet another embodiment of the present invention, the usage monitoring 
appzTBtas can be used to provide targeted surveys to network users. As previously 
discussed an insertion server may be implemmted to send targeted surveys to 

30 network users based upon usage monitoring data. The surveys may be sent to the 
network user unsolicited or the network user may choose to participate in the 
surveys. In fius regard, network users may be selected randomly and a pop-up 
screen will ^pear at randomly selected times on their communication device. See, 
for example, U.S. Patent Application Serial No. 09/541,877, mtitled "Information 
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And Control Console For Use With A Network Gateway Interface" filed April 3, 
2000, in the name of iaventor Short et al., and assigned to the same assignee as the 
present invention. The contents of the '877 application are hereby incoiporated by 
reference as if setforth iiilly herein. 

5 In an alternate embodiment of the present invention, the usage monitoring 

method and apparatus may also monitor queries that a network user submits to a 
network service application, such as a search engine or the like. By monitoring the 
query a network user submits to a network sendee application such as a search 
engine, additional knowledge about the user's preferences can be obtained. Curreut 

10 methods of usage monitoring do not provide for the capability to monitor, in real- 
time, network user queries. Real-time monitoring of network user query strings 
provides network beneficiaries, such as network advertisers and the like flie ability 
to profile the network user and provide the user with more targeted information. 
For mstance, if a user searches for the model Taurus^ on the Ford™ website, a 

15 network advertiser can use this information to direct an advertisement for a 
competing product to the user. ' 

Another form of user input which can be monitored by the usage 
monitoring method and apparatus are the web forms posted by a user, such as 
network service registration forms, i?>plication forms and the like. Current 

20 mettiods do not c^ture information in forms that are submitted by the user on flie 
web. 

As previously stated, the network usage monitoring apparatus and method 
of the current invention provides for the C2?)ability to monitor and store a wide 
spectrum of network user related information. This information includes, but is 

25 not limited to, the contmt of the network service accessed, the demographics of the 
user population (i.e. user population age, gender, geographical location, service 
plan, etc.), and the duration that network services are accessed. In this regard, the 
network service provider or beneficiary will be able to determine real-time 
statistical propaties of users accessing any particular network service at any 

30 particular time. For instance, "How many women between the ages of 34 and 55 

are currently accessing a specific web-site?" or "What proportion of users listming 

to aparticular classical music station are under 35 years of age?" This information 

can then be provided to network beneficiaries, who can exploit this information to 

benefit their core audience in an effective maimer. 
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Main Memory Reqiiirements 

The following discussion serves to hypothetically estimate the amouit of 
main memory required to operate the usage monitoring module in accordance with 
S the present invention. 

Assuming that the average length of an address is / bytes and the number of 
active customers is the amount of stomge can be calculated as 
Sn (Storage in bytes) -1*2 kmax * n 
The worst case storage requirement flien depends on N, the total number of 
10 users that access the network at this particular access point. 

Sn (Storage in bytes) ^l*2knax*N 
ISP (Internet Service Provider) studies indicate that at any time only 10- 
25% of the users are active. As a result, the typical storage requirement Sn is only 
about 25% of the worst case storage requirement Sn- 
1 5 The length of a typical packet is usually under 1 00 bytes though some 

URL's do extend to more than twice this length. The longer URL*s typically result 
fix>m infrequent queries and as a result do not occur very frequently. Thus, for the 
purpose at hand, it can be assumed tiiat the average value of / is 100 bytes. 

The typical length of a user session varies from 55 minutes for dial-15) 
20 customers to 95 nunutes for DSL customers. And since the average time spent on 
a webpage by a user is ^^proximately 2 minutes, it is fair to estimate that the 
number of network addresses visited by any user in one netwoik session is usually 
of the order of 1 00. This gives us an uppo: bound on the value of k. 

Thus, if it is assumed that 128MB of RAM in the primary database is 
25 dedicated to storing navigational sequences identified according to this method and 
the value of is 100 and / is 100 bytes, the usage monitoring module will be 
capable of monitoring nearest neighbors for 12,800 users. One can thus conclude 
that this method scales well in terms of the amount of state required, i.e,, the 
number of preceding and succeeding network addresses that are maintained, and 
30 tiie main memory requirements. 

Secondarv Storage Requirements 

The number of navigational sequences stored in &e primary database will 
typically increase with time as the users visit more of the designated network 
addresses through different navigational sequences. The number also increases 
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with the value of ^ as the navigational sequences themselves become longer. 
There are two ways to reduce the storage required without losing valuable 
information about the navigational sequences. 

■ Aging the sequences: A timestamp is stored with each navigational 
5 sequence. This timestamp indicates the last time the sequence was 

visited. If the sequence has not been visited for a specified time period 
it is timed out and is deleted. 

■ Eliminate infrequent sequences: Zipf s law again indicates that only a 
small proportion of sequences are responsible for most of the accesses. 

10 Thus it is possible to weed out the navigational sequences that are 

visited less than a specified number of times. 

In accordance with an alternate embodiment of the present invention, the 
usage monitoring apparatus and method may be used to capture and store the 

15 neighborhood of designated network addresses. In this embodiment, there is no 
. significance given to tiie order in which network addresses are accessed by the 
user. The only information stored in tiie database is the network addresses fliat 
occur withm a specified number of pages (or time) of the designated network 
address. The number ofnetwork addresses in the neighborhood defined as 

20 usually increases as the user accesses more network addresses. The nranber also 
increases with the value of it. Shnilar to the secondary storage requirements 
defined above tiiere are two ways to reduce the storage required without losing 
valuable information about the neighborhood. 

■ Aging the neighborhood: A timestamp is stored with each network 

25 address entry that is part of the neighborhood. This timestamp indicates 

the last time the network address was visited. If the network service 
has not been visited for a specified time period it is timed out and is 
deleted firom the neighborhood. 

■ Eliminate infrequent neighbors: Zipf s law again indicates that only a 
30 small proportion of network addresses in the neighborhood are 

responsible for a majority of accesses. Thus it is possible to delete the 
network addresses that are visited below a specified number of times 
fix>m the neighboxhood. 
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Just as the previous method describes a way to c^ture the network 
addresses neighborhood of a designated network address; it is also possible to 
capture the domain neighborhood of a designated domain. The advertiser can 
designate the domains that it wants the usage monitor module to monitor. The 
5 usage monitor module then keeps track of the previous k domains that a user has 
accessed at any time. When the user accesses a designated domain the usage 
monitor module stores the previous k domains accessed by the user as well as flie 
following k domains that the user will access. Again it is possible to reduce the 
storage required by aging out domains as well as eliminating infrequently visited 
10 domains. 

Capturing entire navigational sequences of which flie designated network 
address and a specified list of associated network addresses are part of, may be 
considered too storage and computation intensive. In this regard, an alternative 
method is provided for that tracks an associated set of network addresses in the 

IS neighborhood of a designated network address, within a large timeout period T 
(this is to ensure that the storage requirements and state information are reasonable 
and implementable) and with the associated set of network addresses being 
typically defined by the gateway or network administrator. In a network advertiser 
scenario this associated set of netwoik addresses may, for example, belong to 

20 competitors of the designated network address. The network advertiser may wish 
to monitor if prospective customers are visiting a specified list of associated 
addresses (such as the General Motors, Volvo, BMW or Mercedes websites), while 
browsing in the neighboifaood of a designated network address (such as the Ford 
website). The usage monitoring module will provide the C25)ability to search the 

25 navigational sequences, which include the designated network addresses and the 
associated network addresses and provide the advertiser with statistics of 
customers visiting network addresses of associated sites. 

As a result of its position at the edge of the communication network, the 
usage monitoring module according to the present invention can monitor the 

30 requests issued by a pluraUty of users and the responses provided by a number of 
differmt network services in order to collect a canonical set of metrics that is 
usefid for network ssidce providers, network users and network beneficiaries. 
This data can be quite robust and can include web and native streaming content as 
well as user demographics and navigation sequoices. Based upon the robust set of 
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data collected from a plurality of usot who are accessing a wide variety of 
network services and other entities over the Intemet, the usage monitoring method 
and apparatus of the present invention can provide substantial amounts of valuable 
data to network service providers, network users, network beneficiaries and the 
like. 

Many modifications and other mibodiments of tiie invention will come to 
mind to one skilled in the art to which this invention pertains having the benefit of 
the teachings presented in the foregoing descriptions and the associated drawings. 
Therefore, it is to be imderstood that the invention is not to be limited to the 
specific embodiments disclosed and that modifications and other embodiments are 
intended to be included within the scope of the appended claims. Although 
specific terms are employed herein, they are used in a generic and descriptive 
sense only and not for purposes of limitation. 
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THAT WHICH IS CLAIMED: 

1 . An apparatus for monitoring communication network usage 
comprising: 

a usage monitoring module operating at a network access point that 
captures all request packets transmitted from a plurality of network users and all 
5 response packets transmitted from a plurality of network services; wherein tiie 
usage monitoring module filters the captured packets to extract usage monitoring 
data tfaerefit)m; and 

a usage monitoring database in communication with said usage 
monitoring module that receives the filtered usage monitoring data from the usage 
10 monitoring module and stores the data associated with the user request packets and 
network response packets. 

2. The apparatus of Claim 1 , further comprising a gateway device that 
implements the usage monitoring module, wherein the gateway device is disposed 

IS within the network at point of network trafEic aggregation so that it receives all 
requests tarn a plurality of network users and receives all responses fix)m network 
services. 

3 . The apparatus of Claim 1 , wherein the usage monitoring module 
20 filters the cq>tured packets to extract networic addresses and the usage monitoring 

database stores the network addresses. 

4. The apparatus of Claim 3, wherein the network addresses further 
comprise URLs (Uniform Resource Locators). 

25 

5. The ^paratus of Claim 1 , wherein tihe usage monitoring module 
filters the captured packets to extract usage monitoring data comprising at least one 
type of data chosen fixnn the group consisting of user identification, network 
addresses, packet timestamp, referring network address, content-type, content 

30 length, response st^us code and user query string. 
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6. The apparatus of Claim 1, wherein the usage momtoring module 
performs navigational sequencing on captured packets so as to monitor a sequence 
of network addresses accessed by a user. 

S 7. The apparatus of Claim 1, wherein the usage monitoring module 

pCTfonns a status code check on network service response packets to verify that a 
user has access to a requested network service address so tiiat only data associated 
with accessible netwoik services is stored in the usage monitoring database. 

10 8. The apparatus of Claim 1, wherein the usage monitoring database 

further comprises a user request database that temporarily stores user request data 
prior to receiving a response from a requested network service. 

9. The ^paratus of Claim 1 , wherein flie usage monitoring database 
1 5 further comprises a temporary database that temporarily stores user monitoring 

information for network addresses that have only been accessed by a user no more 
than a predetermined TnwiTnimi number of times. 

10. The apparatus of Claim 9, wherein the predetermined miniTymm 
20 number oftimes is one. 

1 1 . The qiparatus of Claim 1, wherein the usage monitoring database 
further comprises a designated network address database fliat stores network 
addresses that have been designated for navigational sequencing. 

25 

12. The apparatus of Claim 1 1, wherein the usage monitoring module is 
adapted to receive the network addresses that have been designated for 
navigational sequencing from a usage monitoring chent 

30 13. The apparatus of Claim 11, wherein the usage monitoring module 

defines &e network addresses that have been designated for navigational 
sequencing based on the frequency that a user accesses &e network address. 
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14. The apparatus of Claim 1, wherein the usage monitoring database 
further con^)rises a primary database that stores the data associated with the user 
request packets and network response packets. 

5 15. The apparatus of Claim 1, further comprising an insertion server in 

communication with the usage monitoring database that provides netwoik users 
with targeted information based on netwoik user usage monitoring data. 

16. The apparatus of Claim 15, wherein the insertion server further 
10 comprises an advertisement insertion servCTlhat provides network users ^ 

targeted advertisements based on networic user usage monitoring data. 

17. The apparatus of Claim 15, wherein the insertion server further 
comprises a survey insertion server that provides network users with targeted 

1 5 surveys based on network user usage monitoring data. 

18. A method for providing usage monitoring in a communications 
network, the method conq)rising: 

capturing all data packets being transmitted from a plurality of users 
20 and a plurality of networic services at a network point of access; 

filtering the captured packets to provide for usage monitoring data; 

and 

storing tiie usage monitoring data in a usage monitoring database. 

25 19. The method of Claim 1 8, wherein capturing all data packets being 

transmitted from a plurality of users and a plinrality of network services at a 
network point of access further comprises capturing all data packets being 
transmitted from a plurahty of users and a plurality of network services at a 
gateway device. 



30 



20. The method of Claim 18, wherein filtering the captured packets to 
provide usage monitoring data further comprises filtering the captured packets to 
extract usage monitoring data comprising at least one type of data chosen from the 
group consisting of user identification, netwoik addresses, packet timestamp, 
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refexring network address, content-type, content length, response status code and 
user query string. 

21. The method of Claim 18, wh^em storing the usage monitoring data 
5 in a usage monitoring database, further conq)rises storing flie usage monitoring 

data in a temporary database prior to storing the usage monitoring data in a 
primary database. 

22. A method for providing usage monitoring in a communications 
network, the method comprising: 

c^turing a transmitted data packet at a network point of access; 
determining if the transmitted data packet is a user generated 
request data packet; and 

storing the transmitted data packet in a first database if the 
transmitted data packet is determined to be a user generated request data packet 

23. The method of Claim 22, wherein cspturing a transmitted data 
packet at a network point of access further comprises capturing a transmitted data 
packet at a network gateway device. 

24. The method of Claim 22, wherein storing the transmitted data 
packet in a first database if the transmitted data packet is determined to be a user 
generated request data packet fiuiher comprises storing the transmitted data packet 
in a first temporary database if the transmitted data packet is determined to be a 
user generated request data packet. 

25. The method of Claim 22, fiirther comprising: 
transferring the transmitted data packet fit)m the first database to a 

second database if the transmitted data packet is matched with an accessible 
30 network s^ce response data packet. 

26. The method of Claim 25, wherein the second database comprises a 
temporary database that stores the transmitted data papket and the matched 
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network service r^onse data packet if a determination is made that the user has 
not exceeded a predetermined number of network service address accesses. 

27. The method of Claim 25, wherein the second database comprises a 
5 primary database that stores the transmitted data packet and the matched network 

service response data packet if a determination is made that the user has exceeded 
a predetermined number of network service address accesses. 

28. The method of Qaim 22, further comprising: 

10 determining if tiie transmitted data packet exceeds a session flow 

limit so as to warrant storage of the first transmitted data packet in tiie first 
database. 

29. A method for providing usage monitoring in a communications 
IS network, the method comprising: 

capturing a transmitted data packet at the network point of access, 
the transmitted data packet being determined to be a network service generated 
response packet; 

verifying user network accessibility of the second transmitted data 

20 packed 

matching &e transmitted data packet with a user generated request 
data packet; and 

storing usage monitoring information &om the matched transmitted 
data packet and user generated request data packet in a database. 

25 

30. The method of Claim 29, wherein capturing a transmitted data 
packet at a network point of access further comprises capturing a transmitted data 
packet at a network gateway device. 

30 31. The method of Claim 29, wherein verifying network accessibility of 

the transmitted data packet further comprises checking the status code of the 
. transmitted data packet to veriiy user network accessibility. 
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32. The method of Claim 29, wherein storing usage monitoring 
information &om the matched transmitted data packet and user generated request 
data packet in a database further comprises storing usage monitoring information 
from the matched transmitted date packet and user generated data packet m a 

5 temporary database if a determination is made that the user has not exceeded a 
predetemoined number of network service address accesses. 

33. The method of Claim 29, wherein storing usage momtoring 
information from the matched transmitted data packet and user generated request 

10 data packet in a database further comprises storing usage monitoring information 
from the matched transmitted date packet and user generated data packet in a 
primary database if a determination is made that the user has exceeded a 
predetermined number of network service address accesses. 

15 34. A method for navigational sequence usage monitoring in a 

communications network, the method comprising: 

designating network addresses requiring navigational sequencing; 
c^turing, at a network point of access, a user transmitted data 
packet associated with a network address; 
20 determining if the network address associated with the captured user 

transmitted data packet is a designated network address; and 

storing a predetermined number of network addresses that are 
accessed proximate the designated network address as a navigational sequence if a 
determination is made that network address associated with the captured user 
25 transmitted data packet is a designated network address. 

35. The method of Claim 34, wherein storing a predetermined number 
of network addresses that are accessed proximate the designated network address 
as a navigational sequence further comprises storing a predetermined number of 

30 network addresses that are accessed subsequent to the access of the designated 
network address as a navigational sequence. 

36. The method of Claim 34, wherein storing a predetermined number 
of network addresses that are accessed proximate the designated network address 
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as a navigational sequence further comprises storing a predetennined number of 
network addresses that are accessed prior to the access of the designated network 
address as a navigational sequence. 



5 37. The method of Claim 34, wherein storing a predetOTnined nimiber 

of network addresses that are accessed proximate the designated network address 
as a navigational sequence further conq)rises stormg a predetennined number of 
network addresses that are accessed prior to the access of the designated network 
address as a navigational sequence. 

10 

38. The method of Claim 34, wherein storing a predetermined number 
of network addresses that are accessed proximate the designated network address 
as a navigational sequence further comprises storing a predetermined nimiber of 
network addresses that are accessed subsequent to and prior to the access of the 

15 designated network address as a navigational sequence. 

39. The method of Claim 34, further comprising: 

detemiining if the user that sent the user transmitted data packet has 
accessed a designated network address in a predetermined number of previous 
20 network address accesses; and 

adding the network address from the user transmitted data packet to 
a navigational sequencing list if a determination is made that the user has accessed 
a designated address in the predetermined number of previous network address 
accesses. 

25 

40. The method of Claim 34, further comprising storing the designated 
network address in a database. 



41 . The method of Claim 34, wherein designating network addresses 
30 requiring navigational sequencing is based upon detemiining the most frequently 
accessed network addresses. 
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42. The meliiod of Claim 34, wherein designating network addresses 
requiring navigational sequencing is based upon network addresses accessed a 
predetermined number of times. 

43. The method of Claim 34, wherein designating networic addresses 
requiring navigational sequmcing is based upon usage monitoring clieat selected 
network addresses. 
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